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APPELLANTS' REPLY BRIEF IN ACCORDANCE WITH 37 C.F.R g 41.41(a) 

Seattle, Washington 
June 13, 2005 

TO THE COMMISSIONER FOR PATENTS: 

This paper is being filed in accordance with 37 C.F.R. § 41.41(a) and in response to an 
Examiner's Answer mailed on April 11, 2005. 

ARGUMENT 

In response to the five sections labeled with the letters (a) through (e) of the Response to 
Arguments portion of the Examiner's Answer beginning at page 15, Appellants submit the 
following responsive arguments. 

a. Appellants 1 "Summary of Balogh et al." Is Accurate 

Appellants disagree with the arguably misleading, abbreviated quotation taken from 
Appellants' "Summary of Balogh et al." at page 13 of the Appeal Brief set forth in italics on 
page 15 of the Examiner's Answer in italics. The full quotation reads as follows: 

Balogh does not disclose retrieving information, including find similar 
clips indicia and keywords, associated with said selected media clip from the 
database, in response to a user selecting a media clip. Balogh also does not 
disclose displaying keywords associated with a selected multimedia clip for 
selection by a user for subsequent searching. Further, Balogh does not disclose 
find similar clips indicia including hidden criteria that identifies and/or groups 
media clips based on human judgment regarding the content of the media clip. 
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The abbreviated quotation appears to imply that the claims are limited to find similar 
clips indicia and keywords that are based on the content of the media clips. Regardless of the 
correctness of this implication, the abbreviated quotation ignores the feature of the claims 
directed to the fact that the find similar clips indicia includes hidden criteria that identifies and/or 
groups media clips based on human judgment regarding the content of the media clips. 

Appellants also disagree with the arguments that are based on the abbreviated quotation. 
First, by stating "images, along with its associated keywords and indicia (all pertinent 
information) are displayed at substantially the same time for possible further query, 11 the 
Examiner's Answer appears to argue that Balogh retrieves information, including find similar 
clips indicia and keywords, associated with a selected media clip from a database, in response to 
a user selecting the media clip, as recited in various ways in the independent claims. 

Although Balogh describes f, [c]aptions that match queries (hits) are returned and 
displayed," the Balogh captions are word or language descriptions, not similar clips indicia, 
much less similar clips indicia that includes hidden criteria that identifies and/or groups media 
clips based on human judgment regarding the content of the media clips. As described in the 
specification of Balogh, the caption is a "description ... of the salient features of the image. The 
caption may be a set of regular English language sentences, as opposed to merely a listing of 
unconnected keywords." Col. 3, lines 26-29. 

The Examiner's Answer also argues that "[t]he query process is iterative, further queries 
can be formulated using previously displayed results as a base." However, the Balogh query 
"iterations" are still word or language queries, as described at Col. 14, lines 51-59: 

From a user's perspective, such iterative querying is implemented by selecting the 
desired image using mouse clicks, and dragging the image into either the 
structured description area or the bibliographic area of the screen. The caption or 
the bibliographic data from the selected image is then copied to form the new 
query, just as if the same information had been input directly by the user, and 
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the user may make further additions, deletions, or modifications to this 
information as desired. 

As described in the section of Balogh quoted above, dragging an image into a description area 
copies description data (caption language) into the new query as if that data had been input by the 
user, and therefore creates a modified word or language query. See also Examiner's Answer, at 
page 18 ("Balogh teaches that the associated criteria/keywords are copied and can be 'further 
edited 1 . . . , a user can then change one keyword while keeping all other criteria the same . . ."). 

In contrast, however, in the exemplary embodiment described in the specification, the 
find similar clips indicia of the claimed invention is invoked by pressing the find similar clips 
button 82, as described in the specification at page 9, lines 22-27 (FIGURE 2A). Upon pressing 
the find similar clips button, a find similar clips interface 90 is displayed, with buttons 92, 94, 
and keywords, for example, 98A and 98B (FIGURE 2B). 

Second, the Examiner's Answer argues that Balogh teaches criteria associated with 
images or media clips, with "associations and suggestions authored by a human user using his/her 
judgment." At most, Balogh teaches that a "captioner" verifies the quality of the image data and 
the bibliographic data, and writes a short "caption," or a description of the salient features of the 
image, and to select certain attributes of the image. See Col. 3, lines 21-29. Col. 3, lines 29-43 
go on to state: 

The attributes may include, for example the type of image (photograph, computer- 
generated graphic, video clip, or other multimedia object, background pattern, 
portrait, abstract, aerial, or special effect), predominant hue, and image orientation 
(landscape or portrait). The captioner also provides as part of the metadata a 
suggest text field describing the emotional suggestions evoked by the image .... 
Collectively, the bibliographic data, the caption, the attributes and the suggests 
field are known as the 'metadata 1 associated with the image. 

As the Examiner's Answer admits, the "caption" that identifies and/or groups media clips 
is not "hidden criteria." See also FIGURE 12; Col. 16, lines 40-62 ("match list command 
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area 1212 also permit the user to alter the display to provide additional information about the 

matches, such as bibliographical information"). 

b. Appellants 1 "Summary of Cox et al." Is Accurate 

The Examiner's Answer disagrees with statements in Appellants 1 "Summary of Cox et al." 
at page 14 of the Appeal Brief. The full quotation reads as follows: 

Cox discloses a queryless multimedia database retrieval method and system that 
maintain a posterior probability distribution for use in selecting the next images to 
display to the user. Cox discloses an entirely software driven selection system. 
No human judgment is involved in the criteria used to identify clips based on 
user selection. 

First, the Examiner's Answer argues that Cox is not a "queryless" method and system. 
The Examiner's Answer defines "query" as "user selection of an image." Ironically, the Cox 
Abstract describes the invention as "[a] queryless, multimedia database search method 
incorporating a Bayesian inference engine that refines its answer with each user response." In a 
nutshell, Cox displays items in a database by capturing a user response to a set of displayed items 
and recomputing the probability distribution over the items in the database according to the user 
response (Claim 1). 

Second, the Examiner's Answer admits that Balogh is relied on for a teaching of "human 
judgment." However, the Examiner's Answer continues to argue that the iterative user selections 
a Cox user makes is also a form of human judgment, because a human chooses selections 
subjectively. While, in the abstract this may be correct, this argument ignores how the term 
"human judgment" is used in the claims. "Human judgment," as used in the claims, pertains to 
the hidden criteria of the find similar clips indicia that identifies or groups media clips, not the 
subjective user response to a set of displayed items, as described in Cox. In Cox, the hidden 
criteria are calculated values representing "feature vectors" (e.g., image width, image, height, 
percentage of pixels that fall into ranges of colorspace, mean saturation, median intensity, 
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contrast, etc., see Table 1) and feature weights (Table 2), which are not criteria based on "human 
judgment." 

c. A ppellants Reiterate, There Is No Motivation to Combine Balogh and Cox to Arrive at 
the Claimed Invention 

The Examiner's Answer provides no basis for correlating that it would be obvious to 
combine Balogh and Cox to arrive at the claimed invention, i.e., the Examiner's Answer 
describes no motivation for combining the teachings of Balogh and Cox. Balogh is generally 
directed to a system and process for retrieving images using a natural language query and English 
language metadata associated with the image. In contrast, Cox is directed to a queryless 
multimedia database retrieval method and system that maintain a posterior probability 
distribution for use in selecting the next images to display to the user. Neither reference teaches 
or suggests why one of ordinary skill in the art would want to combine the teachings of a system 
for retrieving images using a language query with the teachings of a queryless method of finding 
items based on a probability distribution. 

Cox, in fact, disavows query-based methods of searching, like the method of Balogh. As 
stated in Cox at Col. 1, lines 42-47, "query-based methods do not solve the problem of searching 
a multimedia database . . . because (1) multimedia data is generally not annotated with textual 
attribute descriptions, and (2) the vocabulary needed to describe a user's query may not exist or at 
least not be known to the user." Thus, Cox teaches away from, not toward, Balogh. 

d. Balogh Describes a Different Query Development Process 

The Examiner's Answer disagrees with statements in Appellants' "Argument" at pages 18 
and 19 of the Appeal Brief. The full quotation reads as follows: 

Balogh fails to teach or suggest "in response to a user selecting a media clip, 
retrieving information, including find similar clips indicia and keywords, 
associated with said selected media clip from said media clip database." . . . Thus, 
Balogh teaches returning matches based only on words or language entered in a 
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query. In contrast, the present invention teaches beginning with a media clip 
(rather than words or language), and retrieving information "in response to a user 
selecting a media clip." 

The Examiner's Answer argues that "Balogh does not rely solely on words and language, 
but instead relies on the combination of criteria and the media object itself." While Balogh 
describes that a user may use information from the captions retrieved from a first search in a 
subsequent search (Col. 14, lines 40-59), the description data (caption language) is copied into 
the new query as if that data had been input by the user, and therefore creates a new word or 
language query. More specifically, Balogh states as follows (Col. 14, lines 51-59): 

From a user's perspective, such iterative querying is implemented by selecting the 
desired image using mouse clicks, and dragging the image into either the 
structured description area or the bibliographic area of the screen. The caption or 
the bibliographic data from the selected image is then copied to form the new 
query, just as if the same information had been input directly by the user, and the 
user may make further additions, deletions, or modifications to this information as 
desired. 

In contrast, Claim 1, for example, recites: "in response to a user selecting a media clip, 
retrieving information, including find similar clips indicia and keywords, associated with said 
media clip from said media clip database." Elsewhere in Claim 1, the find similar clips indicia is 
recited as "including hidden criteria that identifies and/or groups media clips based on human 
judgment regarding the content of the media clip." Claim 1 further recites that the keywords and 
find similar clips indicia are presented to the user for selection, and selection by the user creates 
search criteria that results in retrieving the media clips that match the search criteria. 

The Balogh process and the claimed invention are entirely different. Balogh's image drag 
and drop action results in the copying of caption or bibliographic data associated with the image 
to form a new query that may be edited by a user. In contrast, the claimed invention provides for 
the display of keywords and find similar clips indicia when an image is selected, the find similar 
clips indicia including hidden criteria that identifies and/or groups media clips based on human 
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judgment regarding the content of the clips. While Balogh suggests that a user can make 
additions, deletions or modifications of the copied data, Balogh does not suggest creating search 
criteria by making selections from displayed keywords and find similar clips indicia, particularly 
find similar clips indicia that includes hidden criteria that identifies and/or groups media clips 
based on human judgment regarding the content of the media clips. 

In summary, Balogh clearly does not disclose displaying keywords or find similar clips 
indicia, much less find similar clips indicia that include hidden criteria associated with a selected 
media clip for selection by a user when creating search criteria. This can occur at the beginning 
of a search or in subsequent search iterations. Balogh merely describes incorporation of caption 
or bibliographic data in a subsequent query. 

This section (d) of the Examiner's Answer includes the following remark: 

Although the rejection states that Balogh does not specifically teach "hidden" 
criteria, nevertheless, Balogh's descriptive captions and keywords suggest a form 
of hidden criteria. 

Frankly, Appellants do not understand this remark. Appellants submit that Balogh's descriptive 
captions and bibliographic data do not suggest a form of hidden criteria. The foregoing remarks 
continue as follows: 

The examiner uses the Cox reference to explicitly teach this feature, however, 
since Cox teaches "true hidden" criteria which is never shown to a user, but is 
nevertheless taught as essential criteria used to identify and judge an image. 

Appellants disagree. While the "hidden" probability distributions of Cox are useful to the extent 

that they produce a relation between displayed items and items being searched for, they would 

have little if any meaning to a user even if displayed. Including hidden probability distribution 

data in the caption of Balogh even if obvious, which Appellants categorically deny, would only 

confuse a user. 
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e. Balogh Fails to Display an Option (for Selection) for Finding Similar Clips 

The Examiner's Answer further disagrees with statements in Appellants 1 "Argument" at 
pages 23 and 24. The full quotation reads as follows: 

Balogh fails to teach or suggest "directly in response to a user selecting a media 
clip from said media clip database, displaying to the user an option for finding 
similar media clips that have an associated find similar clips indicia and/or a 
keyword that matches the find similar clips indicia, human judgment and/or a 
keyword associated with the selected clip." 



Balogh purportedly teaches that information inquiries are processed as in 
conventional keyword searching techniques (Col. 12, lines 9-11), and that a user 
may use information from the captions retrieved from a first search in a 
subsequent search (Col. 14, lines 40-59). However, Balogh does not disclose 
displaying to the user an option (for selection) for finding similar clips, without 
entering a new query with new search language. 

The Examiner's Answer argues that "Balogh teaches this option by providing the optional 
capability of further search refinement via visually dragging and dropping an image, editing, etc." 
Although Balogh arguably provides an "option" for search refinement, Balogh does not teach 
finding similar media clips that have an associated find similar clips indicia and/or a keyword 
that matches the find similar clips indicia, human judgment and/or a keyword associated with the 
selected clip, much less such an option wherein the find similar clips indicia includes hidden 
criteria that identifies and/or groups media clips based on human judgment regarding the content 
of the media clips. 

CONCLUSION 

In view of the foregoing remarks, Appellants submit that all of the claims in the present 
application are clearly patentably distinguishable over the teachings of Balogh et al. and Cox 
et al. taken alone or in combination. In addition to the other differences, as discussed above, 
neither reference teaches or even remotely suggests a "find similar clips indicia including hidden 
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criteria that identifies and/or groups media clips based on human judgment regarding the content 
of the media clip," a recitation included in all of the independent claims on Appeal Therefore, 
Appellants submit that the rejections of Claims 1-9, 11, 12, 14-32, 34, 35, and 37-44 are 
erroneous, and request reversal of these rejections and passage of this application to issue. 

Respectfully submitted, 

CHRISTEN8EN O'CONNOR 
JOHNSON KINDNESS PLLC 
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